Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for the lakectl local --parallelism flag to have persistent default values through env vars/config #8159

Closed
kujenga opened this issue Sep 12, 2024 · 0 comments · Fixed by #8283

Comments

@kujenga
Copy link
Contributor

kujenga commented Sep 12, 2024

This ticket requests that a mechanism is provided to set a default value of the --parallelism flag for the lakectl local commands. Currently, it defaults to 25 and must be explicitly specified on the command line in order to change the default.

It would be fantastic to allow an environment variable and/or config file to set the default value here, similarly to how some of the configs here: https://docs.lakefs.io/reference/configuration.html influence the local CLI, and allow for use of env vars to set them.

Motivation: I have seen a big performance improvement in lakectl local when working with lots of smaller files with an increased --parallelism flag. Rather than needing to remember to set this flag each time, wrap the tool in helper scripts, etc. it would be great if there was a way to provide a higher custom default directly to the tool.

First asked about here: https://lakefs.slack.com/archives/C016726JLJW/p1725922712803829

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants