Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calculate output file md5 hashes and file sizes. #20

Merged
merged 8 commits into from
Sep 19, 2024

Conversation

ajmaurais
Copy link
Contributor

@ajmaurais ajmaurais commented Sep 18, 2024

How the hash file is generated

  • A tsv file is created at the end of the workflow with output file names, paths, md5 hashes, and the file size in bytes.
  • The file sizes and hashes are calculated of all outputs during DiaNN and EncyclopeDIA searches and written to a text file.
  • The hash of the final skyline.zip file is written to a file during the Skyline merge or annotate step
  • The hash txt files are processed and merged into a channel by a new combine_file_hashes sub-workflow.
  • The hashes of additional output files are calculated by a separate process during the combine_file_hashes workflow

Under the hood changes

  • Move container_images.config to conf
  • Define paths used in process publishDir directives to conf/output_directories.config. The output file paths are read by each process and by the combine_file_hashes workflow so that the paths can be included in the hash file.

@mriffle mriffle merged commit 6501e15 into mriffle:main Sep 19, 2024
6 checks passed
@ajmaurais ajmaurais deleted the file_hashes branch September 19, 2024 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants