-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Combine several jobs into one to save computation resources #131
Conversation
Pull Request Test Coverage Report for Build 7511654057
💛 - Coveralls |
Pull Request Test Coverage Report for Build 7511725544
💛 - Coveralls |
Pull Request Test Coverage Report for Build 7583552712
💛 - Coveralls |
@dachengx I quickly went over the code -- just for my understanding: Isn't this the same as just reducing the |
@hammannr I see your concern. Sorry, I was not clear enough in the description. Each job will have a unique |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! Two minor comments then you can go ahead and merge I'd say 😊
alea/submitter.py
Outdated
@@ -387,6 +388,31 @@ def already_done(self, i_args: dict) -> bool: | |||
is_done = False | |||
return is_done | |||
|
|||
def combined_tickets_generator(self): | |||
"""Get the combined submission script for the current configuration. ``self.vcombine_n_job`` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"""Get the combined submission script for the current configuration. ``self.vcombine_n_job`` | |
"""Get the combined submission script for the current configuration. ``self.combine_n_job`` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you maybe add a small example to make clear how/why one should use the combine_n_jobs? 😊
Develop a new ticket generator:
combined_tickets_generator
which combinescombine_n_job
jobs into one by" && "
.Because the initialization of the container and environment will take some time, which is a constant offset proportional to the number of jobs. So this PR will reduce those time.