Feature/speedup parquet export #126
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
add new script which allows exporting to parquet file format. This can be easier to deal with because does not require sql queries to fetch results.
currently not many customizations are supported, just supports the input/output of file.
this allows for easier filtering of the parquet file.
store the osw version and the pyprophet weight table in pandas json format
also clean up the code
Remove IM specific information from export tsv.
Ensure precursor mask has same number of entires as Precursor table in .osw file. Pick the number one ranked feature for precursor mask if possible. (This is not always possible if precursor has no features or if SCORE_MS2.RANK is null for the features) Also add storage for XGB model
remove try/except clause remove argparse for calling script directly
XGB model metadata version metadata
…t into feature/export_parquet
FEATURE_ID in tests are updated to be long integer indicative of what OpenSwath produces
right now not memory efficient to append metadata and bitmasks. Have to fix these later but currently just remove the funcitonality.
Single file multithreading uses a lot less memory
allow for writing temporary files by batches for all cases (not just single file)
score_peptide and score_protein tables missing
fix improper joining occuring on transitions that do not have features
add files and scripts used to create dummy .osw files used for parquet testing
experiment wide context bug fix
Closed
for some reason there are changes to the export and data_handling files which are not used. revoke these changes
|
Currently, this method only supports a combined output file (no split based on runs) and no IPF |
option --only_features allows for only exporting precursors that contain a feature ensure that previous tests still work
|
@singjc can you review this when you get a chance? |
|
Looks great, thank you! Please feel free to merge when you think that it is ready. |
singjc
requested changes
Oct 9, 2024
There was a problem hiding this comment.
Looks great, thanks! Just had some minor comments regarding the FEATURE_{table} score names, if some of these could be retrieved dynamically for cases when new scores are added or old ones removed.
There are some click args that are no longer used/needed. Should probably remove those.
so if more var columns are added in the future they will be parsed automatically
|
@singjc suggestions should be addressed now |
singjc
approved these changes
Oct 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Re-implementation of PR #111
This uses duckdb for parquet export instead of pandas. This allows for memory and SQL queries to be handled by duckdb which is faster and uses less memory.