Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow overriding number of cores used by reconstructors on the CLI #2307

Closed
maxnoe opened this issue Apr 18, 2023 · 1 comment · Fixed by #2430
Closed

Allow overriding number of cores used by reconstructors on the CLI #2307

maxnoe opened this issue Apr 18, 2023 · 1 comment · Fixed by #2430
Assignees
@LukasNickel
Labels
enhancement good first issue

Comments

@maxnoe
Copy link
Member

maxnoe commented Apr 18, 2023

Please describe the use case that requires this feature.

Useful for GRID processing / other job systems with resource management.

Describe the solution you'd like

Add a CLI option that allows overriding the number of cores used by reconstructors e.g. SKLearn based reconstructors to the train and apply tools.

This is needed to be able to set different n_cores for training and application.

Describe alternatives you've considered

Additional context
@maxnoe
Copy link
Member Author

maxnoe commented Oct 21, 2023

I'd propose to add an n_jobs property to the Reconstructor base API. Reconstructors that don't do multiprocessing can just ignore this but reconstructors that are able to use multiple threads must obey this setting.

@LukasNickel LukasNickel self-assigned this Oct 25, 2023
LukasNickel added a commit that referenced this issue Oct 26, 2023
@LukasNickel
Allow setting n_jobs on tool invocation
6c7c270 
- It can be useful to set n_jobs indepently from the config file
- For the sklearn reconstructors this requires setting n_jobs on every
  model, that is attached to the reconstructor
- Fixes #2307
@LukasNickel LukasNickel mentioned this issue Oct 26, 2023
Merged
LukasNickel added a commit that referenced this issue Nov 17, 2023
@LukasNickel
Allow setting n_jobs on tool invocation
cac2366 
- It can be useful to set n_jobs indepently from the config file
- For the sklearn reconstructors this requires setting n_jobs on every
  model, that is attached to the reconstructor
- Fixes #2307
LukasNickel added a commit that referenced this issue Nov 23, 2023
@LukasNickel
Allow setting n_jobs on tool invocation
f209f54 
- It can be useful to set n_jobs indepently from the config file
- For the sklearn reconstructors this requires setting n_jobs on every
  model, that is attached to the reconstructor
- Fixes #2307
Tobychev pushed a commit to Tobychev/ctapipe that referenced this issue Apr 11, 2024
@LukasNickel @Tobychev
Allow setting n_jobs on tool invocation
3cb6aba 
- It can be useful to set n_jobs indepently from the config file
- For the sklearn reconstructors this requires setting n_jobs on every
  model, that is attached to the reconstructor
- Fixes cta-observatory#2307
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LukasNickel LukasNickel

Labels
enhancement good first issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Allow setting n_jobs on tool invocation cta-observatory/ctapipe
2 participants
@maxnoe @LukasNickel