Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge supports analysis files #114

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

Merge supports analysis files #114

wants to merge 15 commits into from

Conversation

lobis
Copy link
Member

@lobis lobis commented Sep 18, 2023

lobis Medium: 108 Powered by Pull Request Badge

This updates the file merging logic so that metadata are also merged.

It is assumed that this is only used to merge files that are generated from an identical configuration file (up to run number, random seed, etc.). The first file metadata will be propagated to the merge file, except for TRestGeant4Metadata that will be the product of the merging of other metadatas (add up number of primaries, etc.).

Metadata, geometry, event and analysis tree will be present in the final file (product of merge).

Originally this merge macro modified the event id of the events if it was duplicated across the new merge file. This is no longer easy to do since we had to use TFileMerger to merge the trees (could not find another way), so there will be duplicated event ids.

@lobis lobis marked this pull request as ready for review September 21, 2023 15:36
@lobis
Copy link
Member Author

lobis commented Sep 22, 2023

So even though the implementation at the time of writing this comment seems to work okay: event, analysis trees are correctly duplicated and metadata are merged, there are some wierd issues when doing the merge where it fails on large number of events after processing many events. Not sure why but I will not merge this until (if) the problem is addressed...

@jgalan
Copy link
Member

jgalan commented Mar 6, 2024

I think doing the merging of analysis tree data is in the scope of TRestDataSet.

@lobis
Copy link
Member Author

lobis commented Mar 8, 2024

I think doing the merging of analysis tree data is in the scope of TRestDataSet.

The original idea of this PR was to have some process that would also merge the metadata. If you just need to merge the analysis tree you can just instantiate a RDataFrame with multiple files as source and do an snapshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants